System and Methods for Calibration of an Array Camera

ABSTRACT

Systems and methods for calibrating an array camera are disclosed. Systems and methods for calibrating an array camera in accordance with embodiments of this invention include the capturing of an image of a test pattern with the array camera such that each imaging component in the array camera captures an image of the test pattern. The image of the test pattern captured by a reference imaging component is then used to derive calibration information for the reference component. A corrected image of the test pattern for the reference component is then generated from the calibration information and the image of the test pattern captured by the reference imaging component. The corrected image is then used with the images captured by each of the associate imaging components associated with the reference component to generate calibration information for the associate imaging components.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.17/653,267 entitled “System and Methods for Calibration of an ArrayCamera” filed on Mar. 2, 2022, which application claims priority as acontinuation of U.S. patent application Ser. No. 17/208,618 entitled“System and Methods for Calibration of an Array Camera” filed on Mar.22, 2021, and issued on Mar. 8, 2022 as U.S. Pat. No. 11,272,161, whichapplication claims priority as a continuation of U.S. patent applicationSer. No. 16/785,914 entitled “System and Methods for Calibration of anArray Camera” filed on Feb. 10, 2020, and issued on Mar. 23, 2021 asU.S. Pat. No. 10,958,892, which application claims priority as acontinuation of U.S. patent application Ser. No. 16/290,672 entitled“System and Methods for Calibration of an Array Camera” filed on Mar. 1,2019, and issued on Feb. 11, 20202 as U.S. Pat. No. 10,560,684, whichapplication claims priority as a continuation of U.S. patent applicationSer. No. 15/987,768 entitled “System and Methods for Calibration of anArray Camera” filed on May 23, 2018, and issued on Mar. 5, 2019 as U.S.Pat. No. 10,225,543, which application claims priority as a continuationof U.S. patent application Ser. No. 14/841,651 entitled “System andMethods for Calibration of an Array Camera” filed on Aug. 31, 2015, andissued on May 29, 2018 as U.S. Pat. No. 9,986,224, which applicationclaims priority as a continuation of U.S. patent application Ser. No.14/519,005, entitled “System and Methods for Calibration of an ArrayCamera”, filed on Oct. 20, 2014 and issued on Sep. 1, 2015 as U.S. Pat.No. 9,124,864, which application claims priority as a continuation ofU.S. patent application Ser. No. 13/792,143, entitled “System andMethods for Calibration of an Array Camera”, filed on Mar. 10, 2013 andissued on Oct. 21, 2014 as U.S. Pat. No. 8,866,912, the disclosures ofwhich are incorporated by reference herein in their entireties.

FIELD OF THE INVENTION

The present invention relates to calibration of an array camera. Moreparticularly, this invention relates to systems and methods forcalibrating each imaging component in an array camera.

BACKGROUND

An array camera is a camera that is made up of multiple imagingcomponents. Each individual imaging component captures data for a twodimensional image of a view. For purposes of this discussion, an imagingcomponent is an individual camera and/or circuitry including an array ofpixels that capture data for a two dimensional image of the view. Thedata of the two-dimensional images used to generate as a light field. Alight field can be used to produce super-resolution (SR) images andother types of images of the view using some or all of the data from thevarious two dimensional images captured by the individual imagingcomponents.

During the manufacture of most digital cameras (including array cameras)and each imaging component of an array camera, a calibration process istypically performed. In a conventional camera or imaging component, thecalibration process typically measures the Modulation Transfer Function(MTF). MTF measurements enable the detection of aberrations that degradeMTF. Some calibration processes may also be used to collect data thatcharacterizes the camera or imaging component in order to adjust theparameters of various image processing algorithms to produce desiredimages. Typically, the need to obtain precise data during calibration isbalanced with the need to keep the manufacturing test overhead to aminimum.

SUMMARY OF THE INVENTION

Systems and methods in accordance with embodiments of this inventionenable calibration of array cameras. In accordance with embodiments ofthis invention, one or more of the imaging components of the arraycamera are designated as a reference imaging component and each of theremaining imaging components in the array camera is an associate imagingcomponent. Each of the associate imaging components is associated withat least one of the reference imaging components.

In some embodiments in accordance with this invention, the calibrationprocess is performed in the following manner. An image of a test patternis captured by each of the imaging components of the array camera. Sceneindependent geometric corrections for the image data captured by thereference imaging component is generated using data of the image of thetest pattern captured by the reference component and data describing thetest pattern using a processor. A corrected image of the test patternfor the reference component is generated based on the scene independentgeometric corrections for the image data captured by the referenceimaging component and the image of the test pattern captured by thereference imaging component using the processor. Scene independentgeometric corrections for the image data captured by each of theassociate imaging components associated with the reference imagingcomponent using the data of image of the test pattern captured by eachassociate imaging component and the corrected image of the referencecomponent using the processor.

In some embodiments in accordance with the invention, the test patternincludes a low-contrast slanted edge pattern. In accordance with some ofthese embodiments, the test pattern includes a plurality of MacbethColor Chart type patterns inset at different positions in thelow-contrast slanted pattern.

In accordance with some embodiments, the test pattern is at a distanceof at least 70 percent of the hyperfocal distance of the array cameraaway from the array camera during the capturing of the image of the testpattern. In accordance with other embodiments, the test pattern is at adistance of at 50 percent of the hyperfocal distance of the array cameraaway from the array camera during the capturing of the image of the testpattern. In some embodiments in accordance with this invention, at leastone pass/fail test of the array camera is performed based on images ofthe test pattern captured by the plurality of imaging components in thearray camera.

In some embodiments in accordance with this invention, the generating ofscene independent geometric corrections for the image data captured bythe reference imaging component using data of the image of the testpattern captured by the reference component and data describing the testpattern is performed in the following manner. The intersection points inthe image of the test pattern captured by the reference imagingcomponent are identified. Uniformity characteristics of the referenceimaging component are determined from the identified intersection pointsin the image of the test pattern captured by the reference imagingcomponent and the test pattern. A set of geometric corrections for thereference imaging component is derived to compensate for low frequencyaberrations in the captured image of the test pattern.

In some embodiments, the generating of scene independent geometriccorrections for the image data captured by each of the associate imagingcomponents associated with the reference imaging component is performedin the following manner. The intersection points in the test patternimage captured by each of the associate imaging components areidentified. The intersection points from the captured test patternimages captured by each of the associate imaging components associatedwith the reference component are translated in accordance with anexpected parallax shift for each of the associate imaging componentsrelative to the reference component. A set of geometric corrections foreach of the associate imaging components associated with the referencecomponent to compensate for low frequency aberrations in the capturedimage of the test pattern by comparing the translated intersectionspoints in the images captured by each of the associate imagingcomponents to corresponding intersection points in the corrected imagefor the reference component. In some of these embodiments, the expectedparallax shift for each of the associate imaging components is basedupon at least one of the physical offset of a particular imagingcomponent to the reference imaging component, the behavior of sensoroptics in the particular associate imaging component, and distance ofthe test pattern from the array camera.

In some embodiments, the images captured by the imaging components arestored in order to perform the calibration of the reference componentand each of the plurality of associate imaging components at a latertime.

In some embodiments, the scene dependent geometric correctioninformation for the image data captured by the reference component isstored in a memory. In some embodiment, the scene dependent geometriccorrection information for the image data captured by each of theassociated imaging components associated with the reference component isstored in a memory.

In accordance with some embodiments, the calibration process furtherincludes generating colorimetric corrections for the image data capturedby each imaging component in the array camera using data of the image ofthe test pattern captured by the each imaging component. In someembodiments, the calibration process further includes generatingphotometric corrections for the image data captured by each imagingcomponent in the array camera using data of the image of the testpattern captured by the reference component using the processor.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 conceptually illustrates an array camera in accordance with anembodiment of this invention.

FIG. 2 illustrates a flow diagram of a calibration process for an arraycamera in accordance with an embodiment of this invention.

FIG. 3 illustrates a test pattern for use in a calibration process foran array camera in accordance with an embodiment of this invention.

FIG. 4 illustrates a flow diagram of a process for calibrating areference imaging component in accordance with embodiments of thisinvention.

FIG. 5 illustrates a flow diagram of a process for calibrating anassociate imaging component with respect to a reference imagingcomponent in an array camera in accordance with embodiments of thisinvention.

DETAILED DESCRIPTION

Turning now to the drawings, systems and methods for performing acalibration process for an array camera in accordance with embodimentsof the invention are illustrated. In accordance with many embodiments ofthe invention, at least one of the imaging components in the arraycamera is designated as a “master” or “reference” imaging component; andeach of the remaining imaging components in the array is an associateimaging component. Each associate imaging component is associated withat least one of the reference imaging components. In severalembodiments, a calibration process is performed to determine relevantcharacteristics of each reference imaging component including (but notlimited to) colorimetric and photometric calibration processes, and/ordetermining scene independent geometric corrections that can be appliedto the image data captured by the imaging component to account fordistortions related to the mechanical construction of the imagingcomponent. The process then determines relevant characteristics of eachof the associate imaging components that are associated with aparticular reference component. In a number of embodiments, thecharacteristics of the associate imaging components are determined withrespect to the corresponding characteristics of the associated referenceimaging component. In several embodiments, scene independent geometriccorrections for the image data captured by an associate imagingcomponent are determined by adjusting the image data captured by theassociate image component for parallax and then comparing the parallaxadjusted image data to a corrected image, where the corrected image isgenerated by applying the appropriate scene independent geometriccorrections to the image data captured by the reference imagingcomponent (i.e. the scene independent geometric corrections for theimage data of the reference imaging component determined during thecalibration of the reference imaging component). Systems and methods forcalibrating array cameras in accordance with embodiments of theinvention are discussed further below.

Capturing Light Field Image Data

A light field, which is often defined as a 4D function characterizingthe light from all direction at all points in a scene, can beinterpreted as a two-dimensional (2D) collection of data from 2D imagesof a scene. Array cameras, such as those described in U.S. patentapplication Ser. No. 12/935,504 entitled “Capturing and Processing ofImages using Monolithic Array camera with Heterogeneous Imagers” toVenkataraman et al., can be utilized to capture light field images. In anumber of embodiments, super resolution processes such as thosedescribed in U.S. patent application Ser. No. 12/967,807 entitled“Systems and Methods for Synthesizing High Resolution Images UsingSuper-Resolution Processes” to Lelescu et al., are utilized tosynthesize a higher resolution 2D image or a stereo pair of higherresolution 2D images from the lower resolution image data in the lightfield captured by the imaging components in an array camera. The termshigh or higher resolution and low or lower resolution are used here in arelative sense and not to indicate the specific resolutions of theimages captured by the array camera. The super resolution processingtechniques described in U.S. patent application Ser. No. 12/967,807 canutilize a variety of calibration information including scene independentgeometric corrections related to the distortions introduced by theconstruction of the imaging components utilized to capture the lowerresolution image data. Scene independent distortions introduced by theimaging components of an array camera can limit the fidelity of a higherresolution image generated by super resolution imaging due touncertainty concerning the relevant location within the scene of a pieceof image data. By measuring the distortions during calibration, theuncertainty introduced by the distortions can be reduced, improving theperformance of the super resolution process. In a similar manner,colorimetric and photometric variation between similar imagingcomponents in a array camera can complicate image processing byincreasing the difficulty of identifying similar points within a scenein the image data captured by the imaging components. The disclosures ofU.S. patent application Ser. No. 12/935,504 and U.S. patent applicationSer. No. 12/967,807 are hereby incorporated by reference in theirentirety.

Each two-dimensional (2D) image in a captured light field is from theviewpoint of one of the imaging components in the array camera. A highresolution image synthesized using super resolution processing issynthesized from a specific viewpoint that can be referred to as areference viewpoint. The reference viewpoint can be from the viewpointof one of the imaging components in an array camera. Alternatively, thereference viewpoint can be an arbitrary virtual viewpoint.

Due to the different viewpoint of each of the imaging components,parallax results in variations in the position of foreground objectswithin the captured images of the scene. Processes for performingparallax detection are discussed in U.S. Provisional Patent ApplicationSer. No. 61/691,666 entitled “Systems and Methods for Parallax Detectionand Correction in Images Captured Using Array Cameras” to Venkataramanet al., the disclosure of which is incorporated by reference herein inits entirety. As is disclosed in U.S. Provisional Patent ApplicationSer. No. 61/691,666, a depth map from a reference viewpoint can begenerated by determining the disparity between the pixels in the imageswithin a light field due to parallax. A depth map indicates the distanceof the surfaces of scene objects from a reference viewpoint. In a numberof embodiments, the computational complexity of generating depth maps isreduced by generating an initial low resolution depth map and thenincreasing the resolution of the depth map in regions where additionaldepth information is desirable such as (but not limited to) regionsinvolving depth transitions and/or regions containing pixels that areoccluded in one or more images within the light field.

During super resolution processing, a depth map can be utilized in avariety of ways. U.S. patent application Ser. No. 12/967,807 describeshow a depth map can be utilized during super resolution processing todynamically refocus a synthesized image to blur the synthesized image tomake portions of the scene that do not lie on the focal plane to appearout of focus. U.S. patent application Ser. No. 12/967,807 also describeshow a depth map can be utilized during super resolution processing togenerate a stereo pair of higher resolution images for use in 3Dapplications. A depth map can also be utilized to synthesize a highresolution image from one or more virtual viewpoints. In this way, arendering device can simulate motion parallax and dolly zoom. Inaddition to utilizing a depth map during super-resolution processing, adepth map can be utilized in a variety of post processing processes toachieve effects including (but not limited to) dynamic refocusing,generation of stereo pairs, and generation of virtual viewpoints withoutperforming super-resolution processing.

Array Camera Architecture

Array cameras in accordance with embodiments of the invention areconfigured so that the array camera software can control the capture oflight field image data and can capture the light field image data into afile that can be used to render one or more images on any of a varietyof appropriately configured rendering devices. An array camera includingan array of imaging components in accordance with an embodiment of theinvention is illustrated in FIG. 1 . The array camera 100 includes anarray camera module 102, a processor 108, a memory 110 and a display112. The array camera module 102 includes an array of imaging components104 formed by a sensor and a lens stack array and the array cameramodule 102 is configured to communicate with a processor 108. Theprocessor 108 is also configured to communicate with one or moredifferent types of memory 110 that can be utilized to store image dataand/or contain machine readable instructions utilized to configure theprocessor to perform processes including (but not limited to) thevarious processes described below. The processor 108 may also beconnected to a network via a network connection 106 to communicate withother processing systems connected to the network. The display 112 canbe utilized by the processor 108 to present a user interface to a userand to display an image rendered using the light field image data.Although the processor is illustrated as a single processor, arraycameras in accordance with embodiments of the invention can utilize asingle processor or multiple processors including (but not limited to) agraphics processing unit (GPU).

In the illustrated embodiment, the processor 108 receives image datagenerated by the array camera module 102 and reconstructs the lightfield captured by the array camera module 102 from the image data.Sensors including multiple focal planes that can be utilized in theconstruction of array camera modules are discussed in U.S. patentapplication Ser. No. 13/106,797 entitled “Architectures for System onChip Array Cameras”, to Pain et al., the disclosure of which isincorporated herein by reference in its entirety. The processor 108 canmanipulate the light field in any of a variety of different waysincluding (but not limited to) determining the depth and visibility ofthe pixels in the light field and synthesizing higher resolution 20images from the image data of the light field.

In the illustrated embodiment, the imaging components 104 are configuredin a 5×5 array. Each imaging component 104 in the array camera module102 is capable of capturing an image of the view. The sensor elementsutilized in the imaging components 104 can be individual light sensingelements such as, but not limited to, traditional CIS (CMOS ImageSensor) pixels, CC0 (charge-coupled device) pixels, high dynamic rangesensor elements, multispectral sensor elements and/or any otherstructure configured to generate an electrical signal indicative oflight incident on the structure. In many embodiments, the sensorelements of the imaging components 104 have similar physical propertiesand receive light via the same optical channel and color filter (wherepresent). In other embodiments, the sensor elements of the imagingcomponents 104 have different characteristics and, in many instances,the characteristics of the sensor elements are related to the colorfilter applied to each sensor element. In several embodiments, thesensor elements of an imaging component includes a plurality of rows ofpixels that also form a plurality of columns of pixels and the pixels ofeach imaging component are contained within a region of the sensor thatdoes not contain pixels from another imaging component.

In many embodiments, an array of images (i.e. a light field) is createdusing the image data captured by the imaging components 104. As notedabove, the processor 108 in accordance with many embodiments of theinvention are configured using appropriate software to take the imagedata within the light field and synthesize one or more high resolutionimages. In several embodiments, the high resolution image is synthesizedfrom a reference viewpoint, typically that of a reference imagingcomponent 104 within the array camera module 102. In many embodiments,the processor is able to synthesize an image from a virtual viewpoint,which does not correspond to the viewpoints of any of the imagingcomponents 104 in the array camera module 102. Unless all of the objectswithin a captured scene are a significant distance from the arraycamera, the images in the light field will include disparity due to thedifferent fields of view of the focal planes used to capture the images.Processes for detecting and correcting for disparity when performingsuper resolution processing in accordance with embodiments of theinvention are discussed in U.S. Provisional Patent Application Ser. No.61/691,666 (incorporated by reference above). The detected disparity canbe utilized to generate a depth map. The high resolution image and depthmap can be encoded and stored in memory 110 in a light field image file.The processor 108 can use the light field image file to render one ormore high resolution images. The processor 108 can also coordinate thesharing of the light field image file with other devices (e.g. via anetwork connection), which can use the light field image file to renderone or more high resolution images.

Although a specific array camera architecture is illustrated in FIG. 1 ,systems and methods for performing calibration of an array camera inaccordance with embodiments of the invention may be performed on anyarray camera configuration appropriate to the requirements of a specificapplication. Systems and methods for performing a calibration processfor an array camera in accordance with embodiments of the invention arediscussed below.

Calibration Process for an Array Camera

An array camera has features beyond those in a conventional camera thatare characterized during calibration to assist super resolutionprocessing algorithms to produce images of a desired resolution. Theinformation obtained during calibration can include a Field of View(FoV) for each of the imaging components in the array and otherinformation that relates one or more imaging components to one another.In many embodiments, the calibration process determines parametersutilized by driver software to adjust the imaging component alignmentdata that drives normalization and parallax compensation processing inpreparation for super resolution processing. Typically, this informationis obtained by using the imaging components in the array to captureimages of specialized charts. These calibration processes often involvesignificant physical space and substantial data management to obtainuseful information. The space and data management requirements can be aproblem in the manufacturing process of an array camera, where bothphysical space and data storage can be at a premium.

To reduce the amount of physical space needed and data storagerequirements for calibration, processes for calibrating an array camerain accordance with many embodiments of the invention involve capturingan image of a test pattern with each of the imaging components in thearray. The captured images can be utilized to calibrate a referenceimaging component, generate a corrected image of the test pattern basedupon calibration information for the reference component, and calibrateeach of the associate imaging components associated with the referencecomponent. In this way, calibration can be performed where thecalibration of the associate imaging components is performed withrespect to the associated reference component using the corrected imagegenerated for the reference component.

A process for calibrating an array camera in accordance with anembodiment of the invention is illustrated in FIG. 2 . The process 200includes capturing an image of a test pattern with the array camera(205). Each imaging component in the array camera captures an image ofthe test pattern. The test pattern has unique properties as discussedbelow with reference to FIG. 3 . In many embodiments, the test patternis placed at a distance farther away than 50% of the Hyperfocal (HF)distance of the array camera. In accordance with other embodiments, thetest pattern is placed at a distance further away than 70% of the HFdistance of the array camera. In accordance with still otherembodiments, the test patent is a placed a distance from the arraycamera equal to the average of the HF distances of the individual lensstacks of the imaging components. In most embodiments, the HF distancewill be in a range of 20-60 cm or 7.9-23.6 inches away from the arraycamera. One skilled in the art will recognize that the exact distancerequirements will depend on the analysis methods used to calibrate theimaging components and the configuration of the array camera. In anumber of the embodiments, the distance between the array camera and thetest pattern is sufficient to produce a captured image of a checkerboardpattern with sufficient sharpness to determine a relative position of anobject in the captured image to a fraction of a pixel. In someembodiments, the position may be determined within 0.5 of the pixelposition or less. In other embodiments, the position may be determinedwithin 0.25 of the pixel position or less. The distance between thearray camera and the test pattern can also be used to determine theparallax shift between the imaging components to aid in determination ofthe relative spatial performance between the imaging components.

After the images are captured, the images captured by each of theimaging components may be stored in a memory and processed locally inreal time or used by an off-line calibration process that can beperformed by another processing system or by the processor in the arraycamera. Furthermore, the images of the test pattern captured by theimaging components in the array camera may be transmitted from theprocessor 108 of the array camera 100 to another processing system via anetwork connection 106 to allow the other processing system to performthe calibration process off-line. Various calibration processes that canbe performed using the captured images in accordance with embodiments ofthe invention are discussed further below.

After each of the imaging components captures an image of the testpattern, some pass/fail tests may be performed (207) to provide grossoperational sorting. The pass/fail tests may include but are not limitedto, an image evaluation to verify proper image capture by the imagingcomponents and/or to enable material sorting. The image evaluation maylook for image levels within expected ranges, identify particularelements in each image such as fiducials (fiduciary markers) or edgedetection in an expected area of the image. Failure to identify expectedelements in a captured image may indicate a gross failure of the imagingcomponent that captured the image. Other types of pass/fail test mayalso or alternatively be performed without departing from thisinvention.

Individual calibration processes may then be performed for each of theindividual imaging components (208). These individual calibrationprocesses may include, but are not limited, to colorimetric andphotometric calibration processes. In accordance with some embodiments,a colorimetric calibration process is performed for each of the imagingcomponents by evaluating Macbeth-style Color Checker patterns in theimage of the test pattern captured by an imaging component. The valuesof these patterns in the image are compared to expected values and anydeviation is identified and stored for use in the processing pipeline ofthe array camera. In accordance with some embodiments, the photometriccalibration is performed in the following manner. The center of theimage is identified based upon the relative brightness in the image. Thecenter typically has the peak brightness of the image. The pixel data isthen fitted to a cosineA4 curve using a least squares method.Photometric correction data is then determined by taking the differencebetween the fitted curve and the desired vignetting profile. Thephotometric correction data is then stored for use in the processingpipeline. Although specific colorimetric and photometric calibrationprocesses are discussed above, any of a variety of colorimetric and/orphotometric calibration processes can be utilized as appropriate to therequirements of specific applications in accordance with embodiments ofthe invention.

Characterization of the image of the test pattern captured by areference imaging component in the array camera (210) is then performedto determine calibration information for the reference imagingcomponent. A reference component is an imaging component in the arraycamera that is used as a reference for the calibration of one or more ofthe remaining associate imaging components in the array. In thedescribed embodiment, the array camera only includes one referencecomponent. However, an array camera may include more than one referenceimaging component. When the array camera includes more than onereference imaging component, the characterization of the image of thetest pattern (210) is repeated for images captured by each of thereference components. Various processes for characterization of theimage of the test pattern captured by a reference component inaccordance with embodiments of the invention are described below withreference to FIG. 4 . The calibration information for the referenceimaging component(s) is then stored (215).

After the image of the reference component(s) is characterized, aprocess for calibrating each of the associate imaging componentsassociated with each reference imaging component can be performed.First, an associate imaging component associated with the referencecomponent is selected (220). The test pattern image captured by theselected associate imaging component is characterized (225) to determinecalibration information for the associate imaging component with respectto the reference imaging component. In many embodiments, the calibrationinformation includes scene independent geometric corrections to accountfor distortions introduced by the mechanical construction of eachimaging component. The calibration information is then stored for theselected associate imaging component (230). The process then determineswhether the array camera includes another associate imaging componentassociated with the reference component (235). If there is anotherimaging component from the array camera associated with the referencecomponent, the process (220-235) is repeated for the next associateimaging component. If there are no other associated imaging componentsassociated with the reference component, process 200 ends. In someembodiments with more than one reference component, calibration ofassociate imaging components (220-235) is repeated for each particularreference component and the associate components associated with theparticular reference component until the calibration process has beenperformed for all of reference imaging components and its associatedassociate components in the array camera.

A test pattern that can be used in a calibration process in accordancewith embodiments of this invention is illustrated in FIG. 3 . The testpattern 300 has a low contrast slanted edge pattern. As shown, the testpattern 300 has a checkerboard pattern providing the low contrastslanted edge pattern. The test pattern 300 changes with a high enoughfrequency to provide a sufficient number of correlation points toidentify an expected level of low-frequency non-uniformities in theoptical systems of the imaging components. Macbeth Color Chart patterns305 are inset at different positions in the test pattern 300 to enablecolor uniformity checks. In accordance with some embodiments, the testpattern 300 may not include other fiducial elements as the slant edgeintersections provide a sufficient number of identifiable points toperform the calibration process.

Characterization of an Image of a Test Pattern

A process for characterization of an image of the test pattern capturedby a reference imaging component to determine the calibrationinformation for the reference component in accordance with embodimentsof this invention is illustrated in FIG. 4 . The process includesidentifying intersections of the test pattern in the captured image(405), characterizing uniformity of the reference camera (410), derivingparameters to compensate for low frequency aberrations (415), andgenerating a corrected version of captured image of the test pattern(420).

The identification of the intersections of the test pattern in thecaptured image (405) is performed by determining the pixel positions ofthe intersections in the test pattern in the captured image using aconventional corner-detection algorithm such as, the Harris cornerdetector or the Shi and Tomasi minimum eigenvalue method. In accordancewith some embodiments, the intersection points are determined tosub-pixel accuracy. In some of these embodiments, the position of theintersection points is determined to an accuracy of 0.5 of a pixelposition or less. In other of these embodiments, the position of theintersection points is determined to an accuracy of 0.25 of a pixelposition or less. In other of these embodiments, position is determinedto an accuracy of 0.1 of a pixel position or better.

The positions of the intersections in the test pattern are then used tocharacterize the uniformity of the reference component (410) included inthe calibration information for the reference component. The uniformityof the reference imaging component is characterized with respect to the“as-designed” performance of the reference imaging component. Thecharacterization is performed by comparing the determined points to anexpected placement based on the test pattern. The comparisons are usedto measure image translation (due to the imaging component), radialgeometric distortion and other residual geometric perturbations.Correction parameters are then derived to compensate for the measureddistortions. In many embodiments the correction parameters are sceneindependent geometric corrections that are applied to image datacaptured in specific pixel locations by the reference imaging component.

The positions of intersection points of the test patterns are then usedin deriving parameters to compensate for low frequency aberrations (420)to include these parameters in the calibration information. Theseparameters include, but are not limited to, X and Y translations tocorrect for imager rotation; a 3^(rd)-order radial translation tocorrect pin-cushion or barrel distortions; and/or a 5^(th)-order radialtranslation to correct mustache or wave distortions. In otherembodiments, any of a variety of parameters can be determined to correctfor any of a number of different distortions as appropriate to therequirements of a specific application.

The calibration information including the characterization informationand compensation parameters are then used to manipulate the capturedimage of the test pattern to generate a corrected image of the testpattern (425). The corrected image can be used to calibrate each of theassociate imaging components associated with the reference imagingcomponent. Process 400 then ends.

Calibrating an Associate Imaging Component

A process for calibrating an associate imaging component using acorrected image of the test pattern derived from the calibration of thereference component to which the associated component is associated inaccordance with embodiments of this invention is illustrated in FIG. 5 .Process 500 includes identifying the positions of the intersections inthe test pattern in the image captured by the associate imagingcomponent (505), translating the positions of identified intersectionpoints to account for the expected parallax shift between the associateimaging component and the reference imaging component (510), andcomparing the translated positions of intersection points to thepositions of the intersection points in the corrected image of the testpattern of the reference component to derive scene independent geometriccorrection data included in the calibration information for theassociate imaging component (515).

The identification of positions of the intersections of the test patternin the captured image (505) is performed by determining the positions ofthe intersections using a conventional alignment algorithm such as thosedescribed above. In accordance with some embodiments, the positions ofthe intersections are determined to sub-pixel accuracy. In some of theseembodiments, the position of the intersection points is determined to anaccuracy of 0.5 of a pixel position or less. In other of theseembodiments, the position of the intersection points is determined to anaccuracy of 0.25 of a pixel position or less. In still other of theseembodiments, the position of the intersection points is determined to anaccuracy of 0.1 of a pixel position or less.

The position of the identified intersection points are then translatedto account for the expected parallax shift between the associate imagingcomponent and the reference imaging component (510). The expectedparallax shift is based upon the sensor design parameters, the behaviorof the particular sensor optics, and calibration test parameters. Thesensor design parameters include, but are not limited to, the physicaloffset of the particular associated imaging component to the referenceimaging component. The behavior of the sensor optics includes, but isnot limited to, the radial distortion of the relative parallax shiftfrom a strictly rectilinear translation. The calibration test parametersinclude, but are not limited to, the distance of the test pattern fromthe array camera when the image is captured.

The comparison of the translated positions of the identifiedintersections for the associate component image to positions ofcorresponding intersection points in the corrected image of thereference imaging component (515) is performed. X and Y offsetinformation for each identified intersection is determined relative tothe position of the intersection in the corrected image. The X and Yoffset information for all of the intersection positions for thereference imaging component can then be used to create scene independentgeometric correction information. The correction information may berepresented by a grid that provides a geometric correction prescriptionfor the pixels of the associate imaging component to derive theappropriate parameters for the associate imaging component. Thecalibration information including the appropriate parameters can then beused during normal operation to correct distortions in the image datacaptured by the associate imaging component.

Although specific processes are described above with respect to FIGS. 4and 5 with respect to determining scene independent geometriccorrections during the calibration or reference and associate imagingcomponents in an array camera, any of a variety of processes can beutilized to determine scene independent geometric corrections asappropriate to the requirements of a specific application in accordancewith embodiments of the invention.

While the above description contains many specific embodiments of theinvention, these should not be construed as limitations on the scope ofthe invention, but rather as an example of embodiments thereof.Accordingly, the scope of the invention should be determined not by theembodiments illustrated, but by the appended claims and theirequivalents.

What is claimed:
 1. A method for synthesizing an image, the methodcomprising: capturing image data using a plurality of imaging componentsin an array of cameras, the image data comprising: a reference imagecaptured by a reference imaging component having a reference viewpoint;and an associate image captured by the associate imaging componenthaving a different second viewpoint; applying corrections to locationsof pixels of the reference image using calibration information for thereference imaging component, the calibration information for thereference imaging component comprising scene independent geometriccorrections for image data captured by the reference imaging componentto account for distortions related to the mechanical construction of theimaging component and produce a corrected reference image; applyingcorrections to locations of pixels of the associate image usingcalibration information for the associate imaging component, thecalibration information for the associate imaging component comprisingscene independent geometric corrections for image data captured by theassociate imaging component that correct locations of pixels in an imagecaptured by the associate imaging component to locations ofcorresponding pixel locations in the corrected reference image, wherecorresponding pixel locations image the same point in a scene in theabsence of disparity due to parallax; generating a depth map bymeasuring disparity due to parallax between the corrected pixels in thereference image and the corrected pixels in the associate image; andsynthesizing an image using the depth map and at least some of thepixels from the captured image data.